Skip to content

fix(clients): replay rate-limited requests instead of surfacing the 429 - #201

Merged
jfrench9 merged 1 commit into
mainfrom
bugfix/retry-rate-limited-requests
Aug 31, 2026
Merged

fix(clients): replay rate-limited requests instead of surfacing the 429#201
jfrench9 merged 1 commit into
mainfrom
bugfix/retry-rate-limited-requests

Conversation

@jfrench9

Copy link
Copy Markdown
Member

Summary

A bulk backfill — a year of history loaded through per-event write calls — runs at the endpoint category budget for minutes at a time, and every rejection came back to the caller as an ordinary failure. A showcase demo tenant load lost 25 of 383 events to exactly this: ~500 extensions writes in ~108s against a 300/min budget, 92% of budget with zero headroom, so ordinary jitter converted straight into dropped rows.

The API answers an exhausted budget with 429 plus Retry-After / X-RateLimit-*, and that rejection is raised by a request dependency before the endpoint handler runs — the rejected call had no effect, so replaying it is safe even for a POST carrying no idempotency key. Nothing else is retried, precisely because nothing else carries that guarantee.

max_retries / retry_delay on RoboSystemsClientConfig have existed since the extensions clients landed but only ever fed SSE reconnects; ordinary HTTP had no retry path at all. They now reach it.

Changes

Hand-written facades under robosystems_client/clients/ and robosystems_client/graphql/ only; generated api/, models/ and client.py untouched.

  • clients/retry.py (new) — RetryingClient, plus backoff_seconds / retry_after_seconds / retrying_authenticated_client.
    • Subclasses httpx.Client rather than installing a custom transport. httpx sets allow_env_proxies = trust_env and transport is None, so a transport-level retry would silently disable environment proxy detection for every caller. Overriding send() leaves proxy, mount, redirect and TLS handling exactly as httpx configures them.
    • Installed through the public set_httpx_client hook, so just generate-sdk cannot wipe it. retrying_authenticated_client reproduces what AuthenticatedClient.get_httpx_client() would have built, including stamping the credential onto the caller's named auth header and leaving timeout unset to match the generated default.
    • Retry-After is applied as a ceiling on the backoff, never as the sleep. The limiter is a sliding window, so the header reports the whole window — the worst case for a caller that filled its budget instantaneously, and minutes of idling for one that merely ran at the sustained rate. Backoff is exponential with full jitter, capped at 30s per attempt.
    • A streamed request body is not replayed: httpx leaves it unread, so the first attempt consumes it and a replay would post nothing. hasattr(request, "_content") is the test.
  • clients/{ledger,investor,document,graph,table,file}_client.py — all 10 AuthenticatedClient construction sites now go through retrying_authenticated_client(..., config=self.config). Function-scoped from ..client import AuthenticatedClient imports that existed only to construct it were swapped for the helper.
  • graphql/client.pyGraphQLClient takes max_retries / retry_delay_ms and posts through RetryingClient, so typed reads get the same treatment as writes. Threaded from self.config at the three GraphQLClient construction sites (ledger / investor / library).
  • tests/test_rate_limit_retry.py (new) — 10 tests against a stub server that rejects a fixed number of requests: replay-until-accepted, the 429 surfacing once retries are exhausted, max_retries=0 sending exactly once, a streamed body not replayed, other statuses untouched, credential and extra headers surviving onto the retrying client, and LedgerClient.create_event_block end-to-end through the real facade and generated op. Plus the backoff unit tests, including that Retry-After: 60 does not become a 60s sleep.

Sync only. set_httpx_client covers the sync_detailed path every facade uses; a caller reaching for a generated asyncio_detailed on one of these clients still gets a plain AsyncClient from get_async_httpx_client().

One incidental fix: the facades previously handed their headers dict to AuthenticatedClient by reference, and get_httpx_client() mutated it in place to add the auth header — so a facade's self.headers (and the shared config["headers"]) silently grew a credential after the first REST call, which then leaked into GraphQLClient. The helper copies instead.

Compatibility

ADDITIVE surface; one deliberate runtime-behavior change, confined to 429.

  • New: robosystems_client.clients.retry (RetryingClient, retrying_authenticated_client, build_httpx_client, backoff_seconds, retry_after_seconds, RETRY_STATUS_CODES, DEFAULT_MAX_RETRIES, DEFAULT_RETRY_DELAY_MS, MAX_BACKOFF_SECONDS); two optional keyword args on GraphQLClient.__init__.
  • Unchanged: every facade signature, return type and error type. No removals, no renames, no model changes.
  • Behavior: a request that would have raised on 429 now waits and retries up to max_retries (default 5) before surfacing the same error. A caller that wrapped these calls in its own 429 backoff will see the delay applied twice; setting max_retries=0 on the config restores the previous behavior exactly. No other status is affected, and a caller who never hits a rate limit sees no change.
  • Stable-tier signatures are untouched, so this ships as a minor. Version bump is the release dispatch's job, not this PR's.

Testing

  • just test-all in-session: ruff check and ruff format --check clean, basedpyright 0 errors, pytest 568 passed / 17 skipped (10 added).
  • The retry path is exercised against a real local HTTP server rather than mocks, including the full generated-op round trip for create-event-block.
  • Not yet run against the failing demo workflow — that needs this published and pulled through robosystems's own lockfile, since the demo clones that repo and uv syncs in the checkout.

A bulk backfill — a year of history loaded through per-event writes —
runs at the endpoint category budget for minutes at a time, and every
rejection came back to the caller as an ordinary failure. A demo tenant
load lost 25 of 383 events to this.

The 429 is raised by a request dependency before the endpoint handler
runs, so the rejected call had no effect and replaying it is safe even
for a POST carrying no idempotency key. Nothing else is retried, since
nothing else carries that guarantee.

RetryingClient subclasses httpx.Client rather than installing a custom
transport: httpx disables environment proxy detection whenever
transport= is supplied, so a transport-level retry would quietly break
proxied callers. It is installed through the public set_httpx_client
hook, leaving the generated api/ layer untouched.

Retry-After is applied as a ceiling on the backoff rather than as the
sleep itself. The limiter is a sliding window, so the header reports the
whole window — the worst case for a caller that filled its budget
instantaneously, and minutes of idling for one that merely ran at the
sustained rate.

max_retries / retry_delay on RoboSystemsClientConfig now reach ordinary
HTTP calls; previously they only fed SSE reconnects.
@jfrench9
jfrench9 merged commit d6bc7ec into main Aug 31, 2026
4 checks passed
@jfrench9
jfrench9 deleted the bugfix/retry-rate-limited-requests branch August 31, 2026 21:28
jfrench9 added a commit to RoboFinSystems/robosystems-typescript-client that referenced this pull request Aug 31, 2026
TypeScript counterpart of RoboFinSystems/robosystems-python-client#201.
A bulk load runs at the endpoint category budget for minutes at a time,
and every rejection came back to the caller as an ordinary failure.

The 429 is raised by a request dependency before the endpoint handler
runs, so the rejected call had no effect and replaying it is safe even
for a POST carrying no idempotency key. Nothing else is retried, since
nothing else carries that guarantee.

createRetryingFetch composes over an existing fetch rather than
replacing it, so the GraphQL client's per-request timeout wrapper stays
in place and each replay gets its own full timeout. Facade REST calls go
through the generated ops, which share the module-level client and take
no per-call transport, so its `fetch` is the only interception point —
installed from the RoboSystemsClients constructor, and never over one
the caller already supplied.

Retry-After is applied as a ceiling on the backoff rather than as the
delay itself. The limiter is a sliding window, so the header reports the
whole window — the worst case for a caller that filled its budget
instantaneously, and minutes of idling for one that merely ran at the
sustained rate.

maxRetries / retryDelay now reach ordinary HTTP calls; previously they
only fed SSE reconnects and operation polling. Set maxRetries: 0 to
surface the rejection immediately, which an interactive surface may
prefer over waiting.
jfrench9 added a commit to RoboFinSystems/robosystems-typescript-client that referenced this pull request Aug 31, 2026
…29 (#207)

## Summary

TypeScript counterpart of RoboFinSystems/robosystems-python-client#201,
for consistency across the two published SDKs.

A bulk load runs at the endpoint category budget for minutes at a time,
and every rejection came back to the caller as an ordinary failure — the
demo tenant load that prompted the Python change lost 25 of 383 events
that way, at 92% of budget with zero headroom.

The API answers an exhausted budget with `429` plus `Retry-After` /
`X-RateLimit-*`, and that rejection is raised by a request dependency
**before** the endpoint handler runs — the rejected call had no effect,
so replaying it is safe even for a `POST` carrying no idempotency key.
Nothing else is retried, precisely because nothing else carries that
guarantee.

`maxRetries` / `retryDelay` have been on `RoboSystemsClientConfig` since
the extensions clients landed but only ever fed SSE reconnects and
operation polling; ordinary HTTP had no retry path. They now reach it.

## Changes

Hand-written `clients/` only; generated `sdk/` and
`clients/graphql/generated/` untouched.

- **`clients/retry.ts`** (new) — `createRetryingFetch`, plus `backoffMs`
/ `retryAfterMs` and the shared constants.
- **Composes over an existing fetch rather than replacing it**, so the
GraphQL client's per-request timeout wrapper stays in place and each
replay gets its own full timeout rather than sharing one across
attempts. The global `fetch` is resolved per call, not captured, so
harnesses that swap `globalThis.fetch` keep working.
- **`Retry-After` is applied as a ceiling on the backoff, never as the
delay.** The limiter is a sliding window, so the header reports the
whole window — the worst case for a caller that filled its budget
instantaneously, and minutes of idling for one that merely ran at the
sustained rate. Exponential with full jitter, capped at 30s per attempt.
- A `ReadableStream` body is not replayed (the first attempt consumes
it), and an aborted signal stops the loop rather than sleeping through
it.
- **`clients/index.ts`** — the `RoboSystemsClients` constructor installs
the retrying fetch on the shared generated client via
`client.setConfig`. That is the only interception point available:
facade REST calls go through the generated ops, which share the
module-level client and take no per-call transport. **A caller who
already supplied a `fetch` keeps it** — we never override an explicit
choice. `createRetryingFetch` and its constants are re-exported from the
package root so consumers reaching for the raw generated SDK can opt in
with `client.setConfig({ fetch: createRetryingFetch() })`.
- **`clients/graphql/client.ts`** — `createGraphQLClient` wraps its
`timeoutFetch` in the retrying fetch, so typed reads get the same
treatment as writes. New optional `maxRetries` / `retryDelay` on
`GraphQLClientConfig`.
- **`clients/{Ledger,Investor,Library}Client.ts`** — optional
`maxRetries` / `retryDelay` on the three facade configs, threaded from
`RoboSystemsClients`.
- **`clients/retry.test.ts`** (new) — 15 tests: replay-until-accepted,
the 429 surfacing once retries are exhausted, `maxRetries: 0` sending
exactly once, other statuses untouched, a streamed body not replayed, an
aborted signal short-circuiting, per-call global resolution, and the
inner fetch re-running on every replay (which is what proves retry wraps
timeout and not the reverse). Plus the backoff units, including that
`Retry-After: 60` does not become a 60s sleep.
- **`clients/index.test.ts`** — the SDK client mock gains `setConfig`
(part of the real surface), and three tests cover the wiring: the
retrying fetch installed, a caller-supplied fetch left alone, and the
budget reaching the GraphQL facades.

## Compatibility

ADDITIVE surface; one deliberate runtime-behavior change, confined to
429.

- **New exports**: `createRetryingFetch`, `RetryOptions`,
`DEFAULT_MAX_RETRIES`, `DEFAULT_RETRY_DELAY_MS`, `MAX_BACKOFF_MS`;
optional `maxRetries` / `retryDelay` on `GraphQLClientConfig` and the
three facade configs.
- **Unchanged**: every facade signature and return type. No removals, no
renames, no narrowed inputs, no generated-type changes.
- **Behavior**: a request that would have failed on 429 now waits and
retries up to `maxRetries` (default 5) before surfacing the same
rejection. Nothing else is affected, and a caller who never hits a rate
limit sees no change.
- **Worth a look from the app owners.** The three frontends are
interactive surfaces, and the default budget means a sustained rate
limit can now sit behind a spinner for up to ~30s instead of erroring
straight away. `maxRetries: 0` restores the old behavior and a lower
value (2 → ~3s worst case) may suit a UI better than the batch-oriented
default. Happy to change the default here if you'd rather it be lower
for browsers.
- Stable-tier signatures untouched, so this **ships as a minor**.
Version bump is the release dispatch's job, not this PR's.

## Testing

- `npm run test:all` (validate → test → build) in-session: prettier,
eslint and `tsc --noEmit` clean, **331 passed** (18 added), build clean.
- `npm run prepare:publish` confirms `artifacts/retry.{js,d.ts,ts}`
lands in the published layout.
- Not regenerated — no API surface change is involved, so `sdk/` was
deliberately untouched.
jfrench9 added a commit that referenced this pull request Aug 31, 2026
…ions status/cancel calls (#202)

## Summary

Two commits, both follow-ups to #201, kept in one PR because they touch
the same file.

**1. The 429 replay reaches the last four REST call sites.** #201
covered every facade that builds an `AuthenticatedClient` — which is
what I grepped for — but `QueryClient`, `OperatorClient` and
`OperationClient` resolve the credential themselves and pass it in
`headers` on a plain `Client`. Cypher queries, operator runs and
operation-status polls each draw on their own category budget, and
polling an operation in a loop is exactly the shape that exhausts one.

This also closes a divergence between the SDKs: in the TypeScript client
every generated op resolves `(options.client ?? client)` against one
shared singleton, so installing the retrying `fetch` covered these by
construction. Python builds a client per call, so each site had to be
reached individually.

**2. `get_operation_status` and `cancel_operation` could not succeed at
all.** Surfaced while writing a test for (1). Both read an attribute off
a model carrying only `additional_properties`:

```
GetOperationStatusResponseGetoperationstatus.from_dict({"status": "completed"}).status
→ AttributeError: object has no attribute 'status'
```

The surrounding `except Exception` then shaped that into a
plausible-looking failure for *every* response — status always returned
`{"status": "error"}`, cancel always returned `False`. Both are
reachable from the facade.

## Changes

### Commit 1 — retry parity

- **`clients/retry.py`** — `retrying_client`, the unauthenticated
sibling of `retrying_authenticated_client`. Same `set_httpx_client`
install; no credential stamping, since these callers supply their own
headers.
- **`clients/query_client.py`** (1), **`clients/operator_client.py`**
(1), **`clients/operation_client.py`** (2) — now build through it.
- **`tests/test_auth_header_resolution.py`** —
`test_status_call_uses_provider_credential` patched the `Client`
constructor, which this routes around. It now asserts the credential on
the client actually handed to the generated op — the behaviour it meant
to pin, independent of how the client is built.

### Commit 2 — the operations response body

- **`clients/operation_client.py`** — `_parsed_dict` reads the body
through `to_dict()`, matching what
`operator_client._poll_for_completion` already does, and both methods
use it.
- **`cancel_operation` had a second defect the first one masked.** Its
SSE cleanup sat *after* an early `return` on the success path, so the
one case that needs the stream closed — the cancel actually landed — was
the case that skipped it. Because the AttributeError meant that `return`
was never reached, the cleanup was dead code outright; fixing only the
accessor would have made it permanently dead on the common path. It now
runs before the outcome is returned.
- **`tests/test_operation_client_status.py`** (new) — 12 tests over both
methods: real status values, a failed operation's error, missing/absent
body, transport failure, cancel true/false, and the stream being closed
on a successful cancel.
- **`tests/test_operation_client_ops.py`** — the pre-existing tests
passed against all of the above because they asserted on bare `Mock`s,
where attribute access always works. They now build the real response
models. Verified they fail with the `AttributeError` when only
`operation_client.py` is reverted.

## Deliberately excluded

- **`auth_integration.py`** — the cookie-based login client. Backoff on
the auth path is a security control, not a convenience.
- **SSE streams** (`sse_client`, `graph_client._wait_with_sse`) — own
reconnect logic.
- **Presigned S3 transfers** (`file_client._http_client`, the
report-bundle download in `ledger_client`) — not our API, not
rate-limited by us.
- **Async** — `set_httpx_client` covers the sync client only. Every
facade is sync, so nothing in-tree hits it.

## Compatibility

ADDITIVE surface. Two runtime-behaviour changes, both on paths that
previously could not work.

- **New export**: `retrying_client`.
- **Unchanged**: every facade signature and return type.
- **Behaviour**: a query / operator / operations call that would have
failed on 429 now retries (`max_retries=0` restores the old behaviour).
`get_operation_status` now returns the real status instead of
`{"status": "error"}`; `cancel_operation` returns the real outcome
instead of always `False`, and closes the stream when the cancel lands.
**A caller that special-cased the broken shapes should be checked** —
anything treating `status == "error"` as "poll again", or
`cancel_operation() is False` as normal, will now see the true value.
- Ships as a **minor**. Version bump is the release dispatch's job.

## Testing

- `just test-all`: ruff, format, `basedpyright` 0 errors, `pytest` **583
passed / 17 skipped** (15 added, 3 rewritten).
- Confirmed no plain `Client(` construction remains under `clients/`
apart from the deliberate `auth_integration` one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant